Finite State Models for the Generation of Large Corpora of Natural Language Texts

Natural languages are probably one of the most common type of input for text processing algorithms. Therefore, it is often desirable to have a large training/testing set of input of this kind, especially when dealing with algorithms tuned for natural language texts. The problem in creating good corpora is that often natural language texts are too short with respect to the dimension required to test effectively the goodness of text processing algorithms, such as string matching and compression algorithms. This is, for instance, the case of the wellknown Canterbury Corpus [RB97], used for testing lossless data compression algorithms, which contains natural language texts with a relative small dimension of not more than 500Kb. The only exception is the “King James Version of the Bible” (approximately 3, 85Mb) contained in the Large Corpus [RB97]. On the other hand corpora of non-textual data contain test files with dimensions up to 3Mb (like the Protein Corpus [NMW99] and the Silesia Corpus [D03]), while testing on random texts is often performed on buffers of dimension 10Mb [ACR99] and 20Mb [CF04]. In many cases the problem due to the lack of big corpus of natural language texts can be solved by simply concatenating a set of collected texts, even with heterogeneous contexts and by different authors. This is the case, for example, of The Linguistic Data Consortium (http://www.ldc.upenn.edu), an open consortium of universities which creates, collects and distributes speech and text databases and other resources for research and development purposes. However, in this context, the task of being able to automatically generate texts which maintain properties of real texts is appealing. In this note we present a preliminary study on a finite state model for text generation which maintains statistical and structural characteristics of natural language texts, i.e., Zipf’s law [Z32] and inverse-rank power law [CF03], thus providing a very good approximation for testing purposes.